Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LiteLLM - support for Vertex AI, Gemini, Anthropic, Bedrock (100+LLMs) #13

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add LiteLLM - support for Vertex AI, Gemini, Anthropic, Bedrock (100+LLMs) #13

wants to merge 1 commit into from

Conversation

ishaan-jaff
Copy link

@ishaan-jaff ishaan-jaff commented Jan 1, 2025 •
edited
Loading

Add LiteLLM - support for Vertex AI, Gemini, Anthropic, Bedrock (100+LLMs)

What's changing

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
LiteLLM is a lightweight package to simplify LLM API calls - use any llm as a drop in replacement for gpt-4o.

Example

from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-cohere-key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="openai/gpt-4o", messages=messages)

# anthropic call
response = completion(model="anthropic/claude-3-sonnet-20240229", messages=messages)
print(response)

Response (OpenAI Format)

{
    "id": "chatcmpl-565d891b-a42e-4c39-8d14-82a1f5208885",
    "created": 1734366691,
    "model": "claude-3-sonnet-20240229",
    "object": "chat.completion",
    "system_fingerprint": null,
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hello! As an AI language model, I don't have feelings, but I'm operating properly and ready to assist you with any questions or tasks you may have. How can I help you today?",
                "role": "assistant",
                "tool_calls": null,
                "function_call": null
            }
        }
    ],
    "usage": {
        "completion_tokens": 43,
        "prompt_tokens": 13,
        "total_tokens": 56,
        "completion_tokens_details": null,
        "prompt_tokens_details": {
            "audio_tokens": null,
            "cached_tokens": 0
        },
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0
    }
}

@ishaan-jaff
Copy link
Author

@aldro61 @AndrewRWilliams can I get a review on this PR ?

@ishaan-jaff ishaan-jaff changed the title add litellm Add LiteLLM - support for Vertex AI, Gemini, Anthropic, Bedrock (100+LLMs) Jan 1, 2025
@ashok-arjun
Copy link
Collaborator

ashok-arjun commented Jan 6, 2025 •
edited
Loading

@ishaan-jaff Thank you for this PR!!

A few things:

  1. Can you also add litellm to the requirements?
  2. If some models require new API keys (such as an Anthropic API key), can you add that to the environment variables list in README.md? Please prepend LiteLLM-specific keys with CIK_LITELLM_ so it'll be easier to distinguish those new keys.
  3. Have you tested if this works, with some LLMs? I'm very curious to know if any specific LLMs' performance stood out.

Thanks again!

@ashok-arjun ashok-arjun self-assigned this Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ashok-arjun ashok-arjun

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ishaan-jaff @ashok-arjun